Skip to content

fix: expose DECIMAL precision and scale in result metadata - #563

Merged
vuanhphung merged 2 commits into
mainfrom
fix/issue-500-decimal-metadata
Nov 16, 2026
Merged

vuanhphung merged 2 commits into
mainfrom
fix/issue-500-decimal-metadata

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Both rows implementations omit Go's optional driver.RowsColumnTypePrecisionScale, so sql.ColumnType.DecimalSize() reports no DECIMAL metadata. Implement it using existing Thrift qualifiers and the retained SEA Decimal128 schema, including zero-row results. Non-decimals still report ok=false; shared type-name/geo mapping stays untouched.

Independent regressions fail before the fix. Verification: make test (1,477 tests, 9 credential-gated skips), make lint, pure/tagged builds, and root/backend race tests pass. Live populated/empty (10,2)/(38,0)/(38,38), DEC/NUMERIC aliases, and non-decimal controls pass on Thrift (1.49s, native decimals off/on) and SEA (1.35s, decimal-as-float off/on). Full column-metadata parity, now including DecimalSize, passes (1.00s). Both metadata tests and the parity guard are selected by nightly CI. The exact shared DATATYPE-043 test passes live without skipping on Thrift (0.614s) and SEA (0.617s). Shared-warehouse timings measure correctness runs, not performance; metadata-only accessors do not affect request/fetch/conversion paths.

Pristine origin/main reproduces the same four CloudFetch subtest failures, 21 tagged-lint issues, and three generated-file gofmt findings. Excluding that single CloudFetch parent, the tagged suite passes (1,641 tests, 47 skips). The shared conformance package has an existing Arrow v12/v18 geo compile mismatch on both branches; its unmodified data-types test runs in isolation.

Cross-driver state

Driver State Code evidence
JDBC Supported DatabricksResultSetMetaData.java:600 @5ad46ee: declared precision/scale getters
Python Supported thrift_backend.py:790, kernel/type_mapping.py:133 @b4cec221: description qualifiers / Arrow decimals
Go Not supported → Supported Optional precision/scale interface; no opt-in, unchanged native-decimal default (off)
Node.js Supported KernelArrowIpc.ts:261 @8ccddd0: manifest qualifiers, Arrow fallback
Kernel Supported producer src/reader/mod.rs:951 @0b5d0edf: precision/scale metadata and empty schemas

Follows declared-schema semantics; unavailable qualifiers return ok=false rather than an exception because Go's interface has no error return. Unlike Node's text fallback, SEA reads Decimal128 directly: the pinned kernel always reconstructs DECIMAL as Decimal128, independently of scan options.

An alternative fix is open in #513. This approach reads the retained schema rather than extending shared type mapping, avoiding the geo workstream, and tests both rows implementations with populated and empty results.

Adversarial review: round 1 converged with no open high findings; seven medium/low findings addressed, and the Thrift conversion-helper refactor deferred to keep the fix scoped.

Fixes #500

This pull request and its description were written by Isaac.

@vuanhphung
vuanhphung marked this pull request as ready for review November 14, 2026 05:55

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung vuanhphung added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Nov 16, 2026
@vuanhphung
vuanhphung enabled auto-merge November 16, 2026 03:59
@vuanhphung
vuanhphung added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 16, 2026
vuanhphung and others added 2 commits November 16, 2026 06:08
Expose the optional precision/scale interface on Thrift and SEA rows using
their existing declared schema. Cover populated and empty results, decimal
boundaries, manifest qualifiers, and non-decimal columns.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Limit SEA qualifiers to the native Decimal128 schema, keep metadata
consistent, and exercise the effective decimal scan options. Include both
live tests and DecimalSize parity in nightly coverage.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung force-pushed the fix/issue-500-decimal-metadata branch from ad453f8 to acf15cf Compare November 16, 2026 06:09
@vuanhphung
vuanhphung enabled auto-merge November 16, 2026 06:10

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Nit

Looks good, with no blocking issues. Both rows implementations return the declared DECIMAL qualifiers safely: on Thrift they come from the TTypeQualifiers I32 values through the existing bounds-checked getColumnMetadataByIndex, and on the kernel backend they come from the retained Decimal128 schema. Non-decimal and missing-qualifier cases return ok=false, and unit, e2e and parity tests cover populated and zero-row results.

Other findings

  • ⚪ Nit — kernelRows.ColumnTypePrecisionScale uses the receiver name resultRows, but every other kernelRows method uses r. Lint won't catch this because .golangci.yml:59 disables ST1016, but using r would match the rest of the file.

@vuanhphung
vuanhphung added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 5ec905b Nov 16, 2026
15 checks passed
@vuanhphung
vuanhphung deleted the fix/issue-500-decimal-metadata branch November 16, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[coverage] Conformance findings: DATATYPE-043

1 participant